A system in which neuron-simulating elements each simulating functions of a neuron, a basic unit for information processing in a living body are built into a network for parallel processing of information is so-called the neuron circuit network (neural network). Such operations as recognition of characteristics, associative storage and movement control are executed in a living body quite easily, but most of the operations can not easily be executed by conventional types of Neumann computer.
For this reason, active efforts are now being made to solve these problems by simulating functions of a neural system in a living body, especially functions specific to a living body, namely parallel processing, self learning or the like through computer simulations or production of prototype hardware dedicated to such purposes.
FIG. 9 shows a model of a neuron (a unit in nerve cell) comprising a section for receiving input from other neurons, a section for converting the input according to certain rules, and a section for outputting the result of conversion. A variable weigh "wji" is assigned to each of the portions coupled to other neurons, and the weigh expresses a degree of coupling. When this value is changed, also construction of the network is changed. Learning in the network means change of the values.
FIG. 10 shows a case where the neurons are built into a network to form a hierarchical neural network. In this figure, A1, A2 and A3 indicate a neuron respectively. Each of the neurons A1, A2, A3 is coupled to a number of neurons like the neuron shown in FIG. 9, and processes signals from the other neurons and outputs result of the processing. A hierarchical network consists of an input layer, an intermediate layer, and an output layer, and there is no coupling in each layer, nor that from the output layer to the input layer. One neuron is coupled to a number of other neurons.
FIG. 10 shows a three-layered network consisting of an input layer, an intermediate layer, and an output layer, but it should be noted that it is also applicable to a multi-layered network having a number of intermediate layers.
Description is made hereinafter for operations of the neurons A1, A2, and A3 shown in FIG. 10 with reference to the neuron shown in FIG. 9 as an example. At first, description is made for a forward process.
In the neural network shown in FIG. 10, a signal inputted into the input layer A1 propagates to the intermediate layer A2, where a forward processing is executed. A result of processing in the intermediate layer A2 propagates to the output layer A3, where a forward processing is executed, and finally an output from the neural network is obtained.
Herein a value indicating a degree of coupling between neurons is called coupling coefficient, and a coupling coefficient for a degree of coupling between a neuron No.j and a neuron No.i is expressed by Wji. Types of coupling are divided to an excitation type of coupling in which output becomes larger as a signal received from a communicating neuron is larger, and a suppression type of coupling in which output becomes smaller as a signal received from a communicating neuron be larger, and Wji larger than zero (0) indicates the excitation type of coupling, while that smaller than zero (0) indicates the suppression type of coupling.
Now assuming that a neuron is a neuron No.j and output from the neuron No.i is Oi, Wji.multidot.Oi which is a product of the coupling coefficient Wji by the output Oi is an input to the neuron No.i. As each neuron is coupled to a number of neurons, so that .SIGMA.Wji.multidot.Oi, which is a sum of inputs to the neurons, becomes an input to the neuron. This is called internal electric potential, and is expressed by the following expression. EQU net.sub.j =.SIGMA.Wji.multidot.Oi (1)
Then a non-leaner processing is applied to this input, and the result is outputted. The function then is called neuron response function, and for instance, the sigmoid function as shown by the following expression (2) is used. EQU f (net.sub.j)=1/{1+exp (-net.sub.j)} (2)
This function is shown in FIG. 11. The value range is "from 0 to 1", and as the input value becomes larger, the value comes closer to "1", while as the value becomes smaller, the value becomes closer to "0". Thus, an output Oj from the neuron j is expressed by the following expression (3). ##EQU1##
Next description is made for a learning function for the neural network above. Brie description is made for a general back propagation algorithm (simply called BP algorithm hereinafter) as a leaning process used in numerical calculation.
In the learning process, when a certain input pattern p is given, a coupling coefficient is changed so that an error between an actual output value and a desired output value will be minimized. An algorithm to obtain the change rate is the BP algorithm.
The error between an actual output value (Opk) of unit k and a desired output value (tpk) thereof is defined as expressed by the following expression (4). EQU Ep=(tpk-Opk).sup.2 /2 (4)
This indicates an error of the unit k in an output layer, while tpk is teacher data given by a man. In learning, strength of all coupling is changed so that the error will be minimized. Actually, the change rate of Wkj when the pattern p is given is expressed by the following expression (5). EQU .DELTA.pWkj.varies.-.differential.E/.differential.Wkj (5)
Using this change rate, the coupling coefficient Wkj is changed. From the expression (4) above, the following expression (6) is obtained. EQU .DELTA.pWkj=.eta..multidot..delta.pk.multidot.Opj (6)
Herein, Opj is an input value from the unit j to the unit k. The error signal .delta.pk varies according to whether the unit k is in an output layer or in an intermediate layer. At first, an error signal .delta.pk in the output layer is as expressed by the following expression (7). EQU .delta.pk=(tpk-Opk).multidot.f'(net.sub.k) (7)
On the other hand, an error signal .delta.pj in an intermediate layer is as expressed by the expression (8) below. EQU .delta.pj=f'(net.sub.j).multidot..SIGMA..delta.pkWkj (8)
It should be noted that f'(net.sub.j) is one-time differentiation of f(net.sub.j), and the detail is described later. Thus, .DELTA.Wji is generally expressed by the following expression (9). EQU .increment.Wji(t+1)=.eta..multidot..delta.pj.multidot.Opi+.alpha..multidot. .DELTA.Wji(t) (9)
So the expression (10) is obtained. Herein t indicates a priority order in learning, .eta. indicates a learning coefficient, and .alpha. indicates a stabilizing coefficient. The first term in the right side of the above expression (9) indicates the .DELTA.Wji obtained through the expression (6) above, while the second term is added to reduce fluctuation in the error and promote convergence. EQU Wji(t+1)=Wji(t)+.DELTA.Wji(t+1) (10)
Computing for the change rate .DELTA.Wji of a coupling coefficient is thus started from a unit in an output layer and then shifted to that in an intermediate layer. Learning advances in a direction reverse to that for input data processing, namely in a backward direction. For this reason, in learning by way of back propagation at first data for leaning is inputted, and the result is outputted (forward process). Strength of all coupling is changed so that an error in the result will be minimized (backward process). Again data for leaning is inputted. This process cycle is repeated until convergence.
The conventional type of hierarchical neural network is a form as shown in FIG. 10. Data flow in forward process in this network is as shown in FIG. 12. This figure shows a case of three-layered hierarchical type of network, and in the forward process input signals Oi1 to Oi4 are given into the input layer (a layer in the left side in FIG. 12), and output signals Ok1 to Ok4 are obtained from the output layer (a layer in the right side in FIG. 12).
On the other hand, FIG. 13 shows a data flow in a leaning process. In the learning process, teacher signals tk1 to tk4 are given to the output layer (a layer in the right side in FIG. 13) to update strength of coupling between each neuron so that outputs from the neural network will be matched to the teacher signals. It should be noted that at present most of operations for the leaning process are executed by an external general-purpose computer.
An example of the network realized with an electric circuit is shown in FIG. 14 (Refer to Japanese Patent Laid-Open Publication No.295188/1987). In this circuit are provided a plurality of amplifiers 4 each having a substantially S-shaped transmittance and a resistance feedback circuit network 2 for connecting output from each amplifier 4 to an input terminal of an amplifier in other layer as indicated by an alternate long and short dash line in the figure. To an input side of each amplifier 4 is connected a CR time constant circuit 3 comprising a grounded capacitor C and a grounded resistor R separately. And input currents I.sub.1, I.sub.2 to IN are inputted to the amplifiers 4 respectively, and an output is obtained as a sum of voltages outputted from the amplifiers 4.
Herein strength of each input or output signal is expressed with a voltage, strength of coupling between neurons with a resistance value of a resistor I connecting I/O lines between the neurons (a lattice point in the resistance feedback circuit network 2), and neuron response function with a transmittance for each amplifier 4. Also as described above, types of coupling are divided to an exciting one and a suppressing one, and mathematically the type is indicated by a positive or a negative sign for a coupling function. However, as it is difficult to indicate positive or negative with any constant in circuit, herein output from the amplifier 4 is divided to two portions (4a, 4b), and one of the two portions is inverted to generate two types of signals, a positive signal and a negative one, and differentiation between positive and negative is realized by selecting the two types of signal. Also an amplifier is used as the sigmoid function f (net) shown in FIG. 11.
Generally when a neural network is constructed with analog circuits, an area of a signal neuron circuit element can be reduced, so that there are provided such advantages as a high integration degree in the neural network or a high execution speed. On the other hand, however, as signal values are expressed with analog values such as a voltage or a current and also computing of each value is executed by an analog element such as an amplifier, there exist dispersion due to the temperature characteristics, and also due to dispersion in a process or manufacturing elements, response characteristics of each element can not be unified, so that output values will disadvantageously become unstable.
In addition, it is difficult to change a value of a coupling coefficient in a neural network through leaning, so that sometimes leaning is executed with an external computer and each coupling coefficient after leaning is downloaded to the hardware to overcome the problem. When this type of method of learning is employed, an additional computer is required, and a learning speed is remarkably retarded.
An example in which a neural network is constructed with digital circuits is shown below. FIG. 15 to FIG. 16 show an example of neural network realized with digital circuits respectively, while FIG. 15 shows an example of a circuit consisting of a signal neuron. In these figures, designated at the reference numeral 11 is a synapse circuit, at 12 a dendrite circuit, and at 13 a cell body circuit.
FIG. 16 shows an example of configuration of the synapse circuit shown in FIG. 15, said synapse circuit provided with a rate multiplier 11b into which a value obtained by multiplying an input pulse f by magnification a (a magnification to be multiplied by a feedback signal; 1 or 2) is inputted, and to the rate multiplier 11b is connected a synapse load register 11c in which a weighed value W is stored. FIG. 17 shows an example of configuration of the cell body circuit 13 consisting of a control circuit 14, a up/down counter 15, a rate multiplier 16, and a gate 17 each connected thereto in this order respectively, and furthermore an up/down memory 18 is provided therein.
In this example, input or output from a neuron unit is expressed with a pulse series, and a quantity of signals is expressed by the pulse density. A coupling coefficient is expressed with a binary number and is stored in the synapse load register 11c.
Computing for signal processing is executed as described below. At first an input signal is inputted into the rate multiplier 11b and a coupling coefficient to a rate value, thus a pulse density for input signals being educed according to the rate value. This operation corresponds to the portion of Wji.multidot.Oi in the expression above for a back propagation model. Also the portion of .SIGMA. in .SIGMA.Wji.multidot.Oi is realized with an OR circuit shown by the dendrite circuit 12. As types of coupling are divided to an exciting one and a suppressing one, they are previously divided to groups, and a logical sum (OR) is computed for each group. In FIG. 15, F1 indicates an exciting output, while F2 indicates a suppressing output. These two types of output are inputted to the up side and down side of the counter 15 shown in FIG. 17 and counted, thus an output being obtained. This output is a binary number, so that the output is converted to a pulse density using the rate multiplier 16. A neural network can be realized by building a network with a plurality of this neuron units. The learning function is realized by inputting a final output from a network into an external computer, executing numerical computation in the computer, and writing the result in the synapse load register 11c to store a coupling coefficient therein.
As described above, when a neural network is built with digital circuits, an area of a signal neuron circuit element becomes larger as compared to that of an analog circuit, so that an integration degree in the network can not be raised, but there is no bad effect such as dispersion in the temperature characteristics or in the manufacturing process thereof, and in addition it is relatively easy to form a circuit with output from a neuron stabilized and the reliability improved.
However, in the case shown in FIG. 15 to FIG. 17, learning by the neural network is executed by using an external computer, so that a computer for learning is required, and a disadvantageously long time is required.
The present applicant already proposed a signal processor based on a neural network built with neurons (Refer to, for instance, Japanese Patent Application No.118087/1993). In the present invention, a signal processor according to this previous application is treated as an example for an embodiment of the present invention, so description is made below for the signal processor based on this conventional art.
In the conventional art for a signal processor, the present applicant proposes a neuron unit using a digital logic circuit therein as a neural circuit network and signal processing with a network circuit using the neuron unit therein.
Herein the basic concept is as described below.
1. All of I/O signals, intermediate signals, and teacher signals in the nerve cell unit are expressed with a pulse string consisting of binary numbers of "0" and "1".
2. A value of a signal in the network is expressed with a pulse density (a number of "1" for a certain period of time).
3. Computing in the nerve cell unit is executed in a form of logical operation between pulse strings.
4. A pulse string for a coupling coefficient is stored in a memory in a nerve cell unit.
5. In learning, an error is computed according to a given teacher signal pulse string, and the coefficient is changed according to a result of computing. In this step, also all of computing for an error as well as for a change of coupling coefficient are executed in a form of logical operation for pulse strings consisting of "0" and "1".
FIG. 18 shows a case of forward processing by a neuron element based on a pulse density system, and the network configuration is like the hierarchical type of neural network shown in FIG. 10.
At first, a signal is converted to binary numbers of "0" and "1", a logical product (AND) of the input Oi expressed with a pulse density and the coupling coefficient Wji is computed for each synapse. This corresponds to Wji.multidot.Oi in the expression (1) above. A pulse density of output from this AND circuit is in terms of probability a product of a pulse density of input signal and a pulse density of a coupling coefficient.
As described above, types of coupling between neurons are divided to an exciting coupling and a suppressing coupling. In case of numerical operation, a sign of a coupling coefficient is treated, for instance, as positive for an exciting coupling and as negative for a suppressing coupling.
In case of pulse density system, types of coupling are divided to an exciting coupling and a suppressing coupling according to whether a sign of the coupling coefficient Wji is positive or negative, and a logical sum is computed through OR operation for each group. This corresponds to a processing for .SIGMA. in the expression 3 as well as to a processing for a non-linear saturation function f (net).
Namely, in computing according to a pulse density, when the pulse density is low, the pulse density as a result of OR operation can be approximated to a sum of pulse densities for OR inputs.
As the pulse density becomes higher, output from an OR circuit is gradually saturated, so that the result does not coincide with a sum of pulse densities, and for this reason non-linearity is generated.
In case of this OR operation, a value P of pulse density is equal to or larger than zero and equal to or smaller than 1 (0.ltoreq.P.ltoreq.1), and furthermore the value changes as a monotonous increase function of a signal level of input, so that the OR operation is like that through the expression (2) or the sigmoid function shown in FIG. 11.
Output from a neuron element based on the pulse density system is "1" only when the OR output net.sub.j.sup.+ from the exciting group obtained through the above operation is "1" and simultaneously the OR output net.sub.j.sup.- from the suppressing group is "0". Namely, the output is as indicated by the following expressions (11) to (13). ##EQU2##
Next description is made for a learning process in the pulse density system. In a neural network where learning is not being executed, an output from the network when a pattern is inputted is not always a desirable one. For this reason, the coupling coefficient is changed so that an error between the actual output value and the desired output value will be minimized by means of a a learning process like in the BP algorithm described above.
[Error signal in an output layer]
At first description is made for an error signal in an output layer. Herein if the error is expressed with a numerical value, the value may be either a positive or a negative one, but as such an expression is impossible in the pulse density system, an error signal in an output layer is defined as expressed by the following expressions (14) and (15), using two types of signal; .delta.k.sup.+ indicating a plus component and .delta.k.sup.- indicating a minus component. ##STR1##
The error signal plus component .delta.k.sup.+ is "1" when the output result Ok is "0" and the teacher signal tk is "1", and otherwise "0".
On the other hand, the error signal minus component .delta.k.sup.- is "1" when the output result Ok is "1" and the teacher signal tk is "0", and otherwise "0".
The error signals .delta.k.sup.+, .delta.k.sup.- correspond to an error (t.sub.pk -O.sub.pk) between a teacher signal and an actual output signal in the expression (7) for obtaining an error signal in an output layer in the BP algorithm described above.
Then, as shown by the expression (7), a product of these error signals and f'(net) which is a one-time differentiation of the output function f(net) is computed to obtain an error signal in an output layer. Generally, for computing an error signal in learning, a differential coefficient obtained by differentiating an output signal with an internal electric potential is required. In the BP algorithm, if a sigmoid function is used as the output function f(net), the one-time differentiation f'(net) is as expressed by the expression (16). ##EQU3##
In the pulse density system, with reference to the expression (16), the differential coefficient consisting of a plus component f'(net).sup.+ and a minus component f'(net).sup.- of the one-time differentiation f'(net) is defined as expressed by the expressions (17) and (18). EQU f'(net).sup.+ =Ok.andgate.Ok (t-1) (17) EQU f'(net).sup.- =Ok.andgate.Ok (t-1) (18)
Herein, Ok (t-1) is a 1 pulse delay value of the output signal Ok. For this reason, a final error signal in the output layer is as expressed by the following expressions (19) and (20). This corresponds to the expression (7) for computing an error signal in an output layer in the BP algorithm described above. ##EQU4##
[Error signal in an intermediate layer]
Also an error signal in an intermediate layer in the pulse density system is obtained with reference to the expression (8) based on the BP algorithm described above. Namely, error signals in the output layer are collected to form an error signal in an intermediate layer. Herein types of coupling are divided to two groups; an exciting coupling and a suppressing one, and a portion of product is expressed with .andgate. (AND), while the portion of sum (.SIGMA.) is expressed with .orgate. (OR).
Furthermore, when computing an error signal in an intermediate layer, cases are divided to four types according whether a coupling coefficient Wkj is positive or negative, and according to whether the error signal .delta.k is positive or negative. At first, in case of exciting coupling, AND of an error signal component .delta.k.sup.+ in an output layer and the coupling coefficient (.delta.k.sup.+ .andgate. Wkj.sup.+) is computed for every neuron in the output layer, and OR for these is computed. This is an error signal plus component .delta.j.sup.+ for a neuron in an intermediate layer (Expression (21)). EQU .delta.j.sup.+ =.orgate. (.delta.k.sup.+.andgate.Wkj.sup.+)(21)
Similarly, AND of an error signal minus component .delta.k.sup.- in an output layer and the coupling coefficient (.delta.k.sup.- .andgate. Wkj.sup.+) is computed for every neuron in the output layer, and OR for these is computed. This is an error signal minus element of a neuron in the intermediate layer (Expression (22)). EQU .delta.j.sup.- =.orgate. (.delta.k.sup.- .andgate.Wkj.sup.+)(22)
Nest description is made for suppressing coupling. In this case, AND of an error signal minus component .delta.k.sup.- and the coupling coefficient (.delta.k.sup.- .andgate.Wkj) is computed for every neuron in an output layer, and OR for these is computed. This is an error signal plus component in an intermediate layer (Expression (23)). EQU .delta.j.sup.+ =.orgate. (.delta.k.sup.- .andgate.Wkj.sup.-)(23)
Similarly, AND of an error signal plus component .delta.k.sup.+ in an output layer and the coupling coefficient (.delta.k.sup.+ .andgate. Wkj.sup.-) is computed for every neuron in the output layer, and OR for these is computed. This is an error signal minus component for a neuron in the intermediate layer. EQU .delta.j.sup.- =.orgate. (.delta.k.sup.+ .andgate.Wkj.sup.-)(24)
Types of a neuron in an intermediate layer and a neuron in an output layer connected thereto are divided to an exciting coupling and a suppressing coupling. For this reason, as a error signal pulse component in an intermediate layer, a logical sum of .delta.j.sup.+ for the exciting coupling in the expression (21) and .delta.j.sup.+ for the suppressing coupling in the expression (23) is computed as an error signal pulse component in an intermediate layer. Similarly, as a error signal minus component in an intermediate layer a logical sum of .delta.j.sup.- in the expression (22) and .delta.j.sup.- for the suppressing coupling in the expression (24) is computed. Namely, the following expressions (25), (26) are applicable. This corresponds to .SIGMA. .delta. pkWkj in the above expression (8) in the BP algorithm. EQU .delta.j.sup.+ ={.orgate. (.delta.k.sup.+ .andgate.Wkj.sup.+)} .orgate. {.orgate.(.orgate.(.delta.k.sup.- .andgate.Wkj.sup.-)} (25) EQU .delta.j.sup.- ={.orgate. (.delta.k.sup.- .andgate.Wkj.sup.+)} .orgate.{.orgate.(.delta.k.sup.+ .andgate.Wkj.sup.-)} (26)
Then, like in the BP algorithm as expressed in the expression (8), .delta.j.sup.+ and .delta.j.sup.- are subjected to processing with the differential coefficient (f'(net)) in the expressions (17), (18). For this reason, the error signals in an intermediate layer (.delta.j.sup.+, .delta.j.sup.-) are obtained through the following expressions (27), (28) respectively. EQU .delta.j.sup.+ =f'(net).sup.+ .andgate..delta.j.sup.+ =Oj.andgate.Oj (t-1) .andgate..delta.j.sup.+ ( 27) EQU .delta.j.sup.- =f'(net).sup.- .andgate..delta.j.sup.- =Oj.andgate.Oj (t-1) .andgate..delta.j.sup.- ( 28)
[Processing with a learning constant .eta.]
Next description is made for processing with a learning constant .eta. in the expression (6) above for obtaining a correction value .DELTA.W for a coupling coefficient according to the BP algorithm. In numerical calculation, as indicated by the expression (6), only simple multiplication of the learning constant .eta. is required, but in case of the pulse density system, the numerical calculation is realized by thinning out pulse strings according to pulse strings as shown below. ##STR2##
Next, a method of calculating a correction rate .DELTA.W for a coupling coefficient through learning is described below. At first, the error signals (.delta..sup.+, .delta..sup.-) in an output layer or an intermediate layer described above are subjected to processing with the learning constant .eta., and furthermore a logical product with an input signal into a neuron (.delta. .eta. .andgate. O) is computed. However, there are error signals .delta..sup.+ and .delta..sup.-), so that the signals are computed through the following expressions respectively to obtain .DELTA.W.sup.+, and .DELTA.W.sup.-. EQU .DELTA.W.sup.+ =.delta..eta..sup.+ .andgate.O (32) EQU .DELTA.W.sup.- =.delta..eta..sup.- .andgate.O (33)
This corresponds to the expression (6) for obtaining .DELTA.W in the BP algorithm. A new coupling coefficient New.PI.W is obtained depending on these values(.DELTA.W.sup.+, and .DELTA.W.sup.-), but the value is treated separately according to whether the coupling coefficient W is an exciting one or a suppressing one. If the coupling coefficient is an exciting one, a component for .DELTA.W.sup.+ is added to and a component for .DELTA.W.sup.- is subtracted from the original W.sup.+. Namely the expression (34) is applicable. ##STR3##
In case of a suppressing coupling, a component for .DELTA.W.sup.+ is subtracted from and a component for a .DELTA.W.sup.- is added to the original W.sup.-. Namely the expression (35) is applicable. ##STR4##
The learning algorithm in the pulse density system is as described above. Now brief description is made for processing flows in a forward process and a learning processing in the pulse density system hierarchical network structure as shown in FIG. 10. At first, in a forward process, when a signal is given to an input layer, the input signal propagates to an intermediate layer, where the signal is processed as signal processing in the intermediate layer through the expressions (11) to (13) above, and the results are propagated to an output layer.
In the output layer, the signals propagated thereto are subjected to processing by the expressions (11) to (13), thus an output as a result of these operations being obtained with the forward process terminated.
In the learning process, after the forward process described above is executed, furthermore a teacher signal is given to the output layer. In the output layer, an error signal in the output layer is obtained through the expressions (19), (20) and the error signal is sent to the intermediate layer. At the same time, this error signal is subjected to the processing with the learning constant .eta. by ways of the expressions (29) to (31), and a logical product with an input signal from the intermediate layer are computed through the expressions (32), (33), and strength of coupling between the output layer and the intermediate layer is changed according to the expressions (34) and (35).
Next as processing in the intermediate layer, depending on an error signal sent from the output layer, an error in the intermediate layer is calculated through the expressions (27), (28), and the result value is subjected to processing with the learning constant .eta. in the expressions (29) to (31), then a logical product with an input signal from the input layer is computed through the expressions (32), (33), strength of coupling between the intermediate layer and the input layer is changed according to the expressions (34), (35), and the learning process is terminated. Subsequently, the learning process is repeated until convergence.
Next description is made for configuration of an actual circuit based on the algorithm described above with reference to FIG. 19 through FIG. 21. Configuration of the neural network is the same as that shown in FIG. 10. FIG. 19 shows a circuit corresponding to synapse in a neuron, while FIG. 20 shows a circuit to compute an error signal in an output layer from a cell body of the neuron in the output layer, output from the output layer and a teacher signal. FIG. 21 shows a circuit in a portion of the neural network for collecting error signals in a cell body of a neuron in the intermediate layer and those in the output layer to obtain an error signal in the intermediate layer. By combining these 3 circuits into a network as shown in FIG. 10, it is possible to realize a neural network circuit based on a digital system which allows self learning.
At first, description is made for FIG. 19. A coupling coefficient for a synapse is stored in a shift register 21. A terminal 21A is a data take-out port, while a terminal 21B is an entrance for data. If this has a function similar to that of a shift register, other ones such as RAM and an address generator may be used. A circuit 22 having the shift register 21 therein is a circuit to execute (Oi .andgate. Wji) in the expressions (11), (12) above, and AND of an input signal and the coupling coefficient is computed. Types of output must be divided to groups according to whether the coupling is an exciting one or a suppressing one, but a higher versatility can be provided by preparing outputs O.sup.+, O.sup.- for each group and switching the output types according to a type of group. For this purpose, a bit indicating whether the coupling is an exciting one or a suppressing one is stored in a memory 23, and a signal switching is performed by a switching circuit 24 using the information. Also as shown in FIG. 20 and FIG. 21, a plurality of gate circuits 31 each based on OR gate configuration and corresponding to a logical sum in the expressions (11), (12) above for processing each input are provided. Furthermore, as shown in the figures, a gate circuit 32 consisting of an AND gate and an invertor which generates output only when the exciting group shown by the expression (13) is "1" and at the same time the suppressing group is "0" is provided.
Next description is made for an error signal. A circuit 34 shown in FIG. 20 is a circuit generating an error signal in an output layer, which is a logical circuit consisting or AND (logical product) and an invertor, and this circuit corresponds to the expressions (19) and (20) above. Namely, the error signals .delta.k.sup.+, .delta.k.sup.- in the output layers are generated from the output Ok from the output layer, the teacher signal tk, and the signal Ok(t-1) generated by delaying the output Ok from the output layer by a shift register 33. Also the expressions (21) to (24) each for obtaining an error signal in the intermediate layer are executed by the gate circuit 26 having the AND gate configuration shown in FIG. 19, thus outputs .delta..sup.+, .delta..sup.- corresponding to the plus component and minus component respectively are obtained. Thus, an error signal varies according to whether the coupling is an exciting one or a suppressing one, so that it is necessary to determine whether the coupling is an exciting one or a suppressing one, and this operation is executed by the gate circuit 25 having AND gate configuration or OR gate configuration respectively according to whether a bit stored in the memory 23 is for an exciting coupling or a suppressing one, and also according to whether the error signal is .delta..sup.+ or .delta..sup.-. Also the expressions (25), (26) each for collecting error signals is executed by a gate circuit 36 having OR gate configuration shown in FIG. 21. The circuit executing the expressions (27), (28) for obtaining error signals .delta.j.sup.+, .delta.j.sup.- respectively is the circuit 37 shown in FIG. 21, where and the error signals .delta.j.sup.+, .delta.j.sup.- are generated from the signal Oj(t-1) obtained by delaying the output Oj from the intermediate layer with the shift register 33 and an error signal from the output which is an output from the OR circuit 36. Also the expressions (29) to (31) corresponding to a learning rate are executed by the .eta. circuit 35 for a learning constant shown in FIG. 20 and FIG. 21.
At last description is made for a portion in which a new coupling coefficient is calculated from an error signal. This operations are expressed by the expressions (32) to (35) above, and the operations are executed by the gate circuit 27 consisting of AND, invertor, and OR gates shown in FIG. 19. Also type of coupling in this gate circuit must be classified to an exciting one or a suppressing one, and this classification is executed by the gate circuit 25.
In the neural network based on the pulse density system according to the conventional art, it is considered that the method of obtaining a differential coefficient for an output function is not applicable to the expressions (19), (20) each for obtaining an error signal in the output layer in the learning process for updating the coupling coefficient, nor to the circuit for executing the operation (shown in FIG. 20).
For, in the expressions (19), (20) for obtaining error signals .delta.k.sup.+, .delta.k.sup.- in an output layer respectively, and especially in the expression (19) for obtaining .delta.k.sup.+, as shown in the expression, a negating component for Ok is duplicated, and "a logical product of negation of Ok and negation of Ok" is abbreviated to "negation of Ok", while in the expression (20) for obtaining .delta.k.sup.-, the OK component is duplicated as shown in the expression, and "a logical product of Ok and Ok" is abbreviated to "Ok". In a learning process in a neural network, learning can not be executed successfully due to this effect, and there occur such problems as low convergence rate.